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The Crisis in Measurement Literacy in Psychology and Education 



Nadine M. Lambert 



University of California, Berkeley 



The topic of this address was prompted by many experiences and 
published commentary over the past decade that raise serious questions about 
our success as measurement specialists in translating our ever increasing 
sophistication m psychometric theory to the psychological and educational 
publics who use our various products, as weU as the extent to which we have 
been successful in sustaining and nurturing instruction in measurement 
theory and in our many institutions of higher learning. 

What Is "measurPHient literapy*? 

To have achieved minimal literacy in measurement, as a psychologist, 
an educator, a personnel manager, a counselor, or other human service 
practitioner, one would have acquired at least four types of knowledge: 

1) Knowledge of the basic assumptions that underlie the rendering or 
quantification of observations, assignmg objects or events to classes, ordering 
of units of observations from greatest to smallest,' or transforming of the 
number of right and wrong test answers on a formal or informal test. 

2) Familiarity with the general rules by which observations, rank 
orders, item scores, and individual difference data are translated into 
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measurement units such as frequency counts, probabilit>' estimates, measures 
of central tendency, and measures of variability. 

3) Measurement literacy implies familiarity with concepts of validity 
and reliability and the ability to utilize these concepts in selecting, using, or 
interpreting numbers that are derived from the several approaches to 
educational and psychological measurement. 

4) And fourthly knowledge of sources of error and the ability to apply 
appropriate standard errors of measurement in making psychological or 
educational diagnoses, classifications, inferences, or predictions. 

Is there a crisis in measu rement lifprp ^ ^ 

The crisis in measurement Hteracy is evident in several professional 
settings. Even though accreditation standards require courses in 
measurement, students often are not exposed to instruction in psychometric 
theory as part of their programs. Courses in individual testing, program 
evaluation, or statistics are assumed as satisfying the measurement standard. 
Psychologists on accreditation panels observe frequently, and express concern 
about the fact, that psychology and psychologists are losing then- 
measurement foundation. 

Members of departmental faailty who have argued for sustaining or 
increasing the number of faculty positions for psychometric specialists 
encounter considerable departmental resistance to their justifications of the 
importance of adding or retaining a position as a professor in measurement 
when few courses are offered or taken by undergraduate and graduate 
students m both psychology an iucation units. If departments don't have a 
measurement curriculum, they don't need psychometric faculty. 

Officials on psychology licensure boards who review academic 
credentials of applicants for psychology Ucenses note that even though the 
licensure standards require at least some instruction in measurement theory, 
that such instruction is subsumed by courses in assessment or testing rather 
than courses in psychometric theory. 

The perception of many people, including otherwise credible 
professionals in education and psychology, that the elimination of 
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standardized tests will eliminate bias in personnel selection and higher 
education selection, is a telling comment on the lack of clear thinking that is z 
necessary part of literacy 

Those of us who are involved with educational uses of tests in 
elementary, secondary, or higher education typically assume incorrectly thai 
teachers, school administrators, and other education professionals have 
sufficient knowledge to select and use tests that satisfy, at least minimally, the 
Standards for Educati onal and Psycholog ical Tpsf:;. 



The demand for meas urement Tifprary 
Test User Qualifications 

We are indebted to the Test User Qualification Working Group for 
producing a model test user qualification system (Eyde, Moreland, Robertson, 
Primoff, and Most, 1988) to serve as a tool for identifying competent test users 
regardless of educational background or title. The current state of 
measurement Uteracy is evident from their analysis of the content domains 
of test user qudifications, and the types of test misuse for various types of 
tests. 

They identified 86 generic subelements, or knowledges, skills, abilities, 
and other characteristics which were relevant to preventing misuse of the 
more than 50 sample commercial tests used in the test user study. Next they 
proceeded to identify both the minimum essentials or subelements of good 
test use and the more comprehensive requirements for engaging in good 
testing practices Minimum essentials included "keeping scoring keys and test 
materials under close scrunny", and"not making photocopies of copyrighted 
materials," Examples of more sophisticated knowledge were the necessity to 
compare test scores with other data in a psychological history, bemg able to 
use the standard error of measurement, appropriate consideration of 
measurement error in using cutoff scores, and being able to select and use 
norm reference scores appropriately. 

In order to guide test purchase by reference to user qualifications, 8 
empirically derived clusters of tests were identified. These were classified as I; 
group educational tests, 2) ability and preference tests, 3) learning disability 
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and neuropsychological tests, 4) individual intelligence tests, 5) readiness 
tests, 6) objecdve personality tests, 7) self-administered and self-scored tests, 
and 8) projective tests. Their research, therefore, provides an excellent 
framework for setting standards for measurement literacy in the various 
professional practice domains where particular types of tests are used Each of 
these dusters of tests can be matched with groups of psychological and 
educational professional specializations. For example, group educational tests 
are the province of the teacher and education professional. Objective and 
projective personaUty tests are used primarily by cUnical and counselling 
psychologists, although I have read reports of the use of graphology in 
personnel selection. 

The specification of test user competencies resulting from the efforts of 
the Test User Work Group would be especially useful to the APA Committee 
on Accreditation, as they evaluate the measurement competencies provided 
in a program's professional preparation curriculum. Moreover, state 
Hcensure boards and those who prepare examinations for licensure could use 
these findings to evaluate the evidence of measurement competence 
submitted by an individual who wishes to be Ucensed to practice 
independently and who will be a future test user. In turn, department chairs 
and faculty can use the test user competency model as a basis for developing 
and evaluating current and future measurement instruction. 

Before the theoretically-oriented psychometridan argues that 
promoting test user competency is a problem to be solved by someone else, it 
is important to keep in mind that by providing guidelines for test user 
competence, and relating these guidelines to program requirements for 
accreditation or approval, one creates a need and increases the demand for 
measurement instruction in our colleges and universities. As the needs 
become evident, and demands for help increase, campus administrators and 
personnel committees will respond by securing faculty resources for 
measurement instruction, research, and theory development. 

Multiple perspecrives on teachers and testing 

Lambert ( 1980 ) reported the results of a survey of representatives of 
teacher unions, deans of schools of education, and legislators regarding 
opinions of tests and testing. The most frequent answers from all three 
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groups of respondents about teacher attitudes toward tests and testing were 
"negative, afraid of results, suspicious, a threat to job security." The majority 
of deans and legislators felt that teachers' attitudes toward tests should be 
changed and they suggested that: 1) teachers learn more about existing tests 
and how to interpret them; 2) teachers accept tests as ijseful measures, and; 3) 
teachers not rely on tests as the sole source of information about performance 
differences. 

Nearly aU of the respondents to the survey stated that it was important 
for teachers to produce superior classroom tests. And responses to questions 
about criterion-referenced tests suggested that although these might provide 
alternatives to nationally standardized tests, teachers were judged not to 
understand the concepts underlying criterion-referenced measurement. 

The results of the survey showed that demand for measurement 
literacy of teachers was generally recognized; however, of the 102 deans of 
schools of education who responded to tlie survey, only one quarter of them 
said that instruction in testing was offered in a 3-4 semester hour course. 
Another one third provided measurement training as a segment of another 
course, and one fourth of them stated that they had no intention of offering 
any instruction in measurement. 

Although this is only one set of data on teacher attitudes about tests, 
and the need for teacher measurement competency, the results reflect 
widespread support for measurement literacy among teachers on the part of 
teacher organizational leaders, heads of teacher training institutions, and our 
legislators. 

Measurement T.itPrary in the PracHce of Psvchnlngy 

The Committees appointed lo draft the several documents providing 
policy and guidance for the use of tests in academic, applied, and professional 
psychology have worked diligently to produce documents that can be 
understood by a broad range of test usere. But when APA Council 
Representatives and committee and board members, become involved in 
governance review of such documents as the Standards for Educational and 
Psychological Tests, (APA, 1985) the Code of Fair Testing Prartirps m 
Education , (Fremer, Diamond, and Camara, 1989) the Guidel ines for 
Computer-Based Tests and Interp retation (APA^ 1986), or the proposals for 
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test user guidelines, they soon recognized that the difficulty level of the 
concepts defined in the Standards far exceeded the understanding of many 
psychologists. To resolve this dilemma, the APA Council of Representatives 
appointed a special review committee composed of a variety of academic and 
professional psychologists who were themselves not measurement experts to 
review the final prepublication draft of the Standards to review and to 
propose language for the Standards that the typical test user could 
understand. 

The experience of those reviewing the Standards provided eloquent 
testimony to the failure of our doctoral programs for insuring minimal 
competence in measurement among psychologists generally. Criteria for 
recognition of doctoral programs and doctoral preparation for professional 
practice for psycholology have specified the need for basic instruction in 
measurement theory for many years. Both the Criteria for Accreditation 
(APA 1979) as well as standards for licensure (American Association of State 
and Psychology Boards and the Professional Examination Service, 1984) 
require knowledge of basic measurement principles as distinct from 
preparation in research methodology and research design. The examination 
for licensure candidates also includes extensive sections on measurement. 
One strategy for addressing measurement literacy among psychologists and 
educators is to analyze item performance data on measurement items on 
licensure and teacher competency examinations. As the item data ft-om these 
examinations will now be made available, it should be possible to obtain 
empirical evidence on the proportion of licensure and credential candidates 
who perform at an acceptable level on questions pertaining to basic knowlege 
of measurement theory and technique. And if psychometricians are critical of 
the coverage of measurement content in the licensure examination, the 
various state Hcensure and credentiallmg boai-ds, as well as the American 
Association of State Psychology Boards, would be responsive to suggestions as 
to how the content of the examination can be improved. And as 
requirements for measurement knowledge in preprofessional programs 
become more expKcit and regulated, the demand for measurement 
instruction in psychology departments and professional psychology programs 
will follow suit. 
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The SWPIv side of the measnrempnf l iteracy g nPfiH^i^ 

PromoHng a complpmpnt of measure ment faculty and measurpmpnf 
courses in university programs. It seems pretty obvious that the number of 
faculty positions with measurement as a primary field of study, as well as the 
number of students interested in measurement as a careei has been generally 
declining. As Davison, Damrin, and Drasgow (1986) concluded from thek 
analysis of the number of doctoral programs in measurement and the 
number of graduate students, while the number of programs (661 remained 
relatively constant from 1973 to 1983, the proportion of programs has not 
matched the growth in psychology programs over the same period and 
dropped from .18 to .11. The recent study by Aiken, West, Sechrest, and Reno 
(1989) reported that one third of psychology departments offered no training 
in measurement. Although nearly all departments provided courses in 
statistics, only 17% of psychology departinents offered a specialization in 
quantitative areas. Over all of the departments surveyed, only 108 students 
were enrolled in programs emphasizmg quantitative methods, and of these 
only a portion were in the measurement area. 

The Aiken et al. study forecasts tite increasingly common challenge 
that will have to be faced when measurement faculty retire and departinent 
heads have to secure replacements. A case in point is the relatively recent 
retirement of a measurement insti-uctor in my own instihition. As head of 
the Division of Educational Psychology, I prepared several memoranda 
presenting arguments for retaining the position. The administi-ation 
countered titat there were few applicants for the measurement program, that 
measurement could be taught by statisticians in otiier departinents, and that a 
professor holding th« position would not be perceived as providing courses 
for which there wouid be a high stiident demand generally. 

Faced v -^h this challenge, I responded that the state of measurement 
literacy among educators was at an appallingly low level, and that schools of 
education aiming to forge new directions in the preparation of graduate 
students for service to schools and agencies required experts in measurement 
theory as much as they required experts in computer technology. 
Measurement theory, I argued, was a more fundamental area of instiiiction 
in graduate programs than was insfanction in computer applications to 
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teaching. To counter the argument that other faculty could carry the 
measurement courses, I pointed cut that psychometrics was a field of its own, 
and only an appropriately prepared psychometrician would be expected to 
have a research program that would support instruction in classical and 
modem test theory and their applications to common psychological and 
educational problems. 

By pointing out that our success in recruiting students to our applicant 
pool was dependent on visible measurement faculty, as well as a high level 
measurement curriculum, I hoped to neutralize the argument with which I 
was faced, that student demand should be the criterion for faculty resource 
allocation. So it was necessary to argue for the wide applicablUty of 
measurement instruction to program evaluation, teacher preparation, 
preparation of psychologists to work in schools, as well as for many students 
in education whose own research required attention to measurement 
fundamentals. After we won approval for recruiting a measurement faculty 
member, the search and selection process taught us another important lesson 
- that the number of psychometricians with promising psychometric research 
programs interested in a faculty appointment rather than one with a 
psychometric research institute or testing company was small indeed. 

Increasing the demand f or measurpment literacy anH 
promoting the gwplv of mea surement sp eriaTi>tff 

Our field is faced with a genuine supply and demand problem. 
The number of students entering measurement programs is limited by the 
number of available measurement faculty, and in turn the number of 
measurement courses and perception of need for measurement competence 
affects the ability to argue for faculty measurement posidons. If one examines 
the faculty recruitment efforts in any academic year, the advertised 
measurement positions require a person who has hybrid credentials - 
measurement and statistics, or measurement and personalit>' psychology, or 
measurement and social psychology. Nationwide, only a few of the major 
research universities have programs in measurement, but nearly all 
universities have programs for professional educators and psychologists. 

There are several strategies for action .or preserving measurement 
courses progranis, and student bodies. One strategy is tD capitalize on the 
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current popular methodologies such as "qualitative" methods and show how 
measurement theory can be applied to the technical problems encountered in 
rendering qualitative data into valid and reliable measures. The reform 
movement in education raises some other strategic possibilities. As 
professionals have shied away from tests with potential bias, and in turn 
moved toward an egalitarian or anti-individual difference value system, we 
now are faced with tiie challenge of developing new assessment 
methodologies to reflect cognitive and social research findings supporting 
interactive assessment, cognitive modifiability and social and/or cooperative 
learning. But to succeed in these efforts the measurement specialists w.il 
have maintain an involvement with current and fuhire educational and 
psychological faculties as well as with future test users. 

Proposals for increasinf^ demand for m e asurement instruction anH 
measurement litera cy 

Joint Committee on Testing Prarfirpc is an interorganizational 
group representing APA, AERA, MCME and otiier organizations. It 
participates in various assessment and measurement projects, such as the 
lestUserj^ualifications Working Croup The recently published Code of Fair 
Testing Practices in Education is another example. The Joint Committee 
originated from a conference sponsored by APA in the Summer of 1984 in 
which test publishers and psychologists came together to develop a consensus 
on what the pro..iems in testing were, and to develop an agreed upon 
approach to address particular problems in a cooperative way. One strategy 
for promoting measurement literacy is to support actively the 
implementation of the products that the Joint Committee develops both by 
working v.'lth publishers, as well as to provide reference sources for 
reviewing content of measurement courses in our university programs. 

In addition to becoming aware of tiie potential of the several products 
of the Joint Committee on Testing Practices and utilizing them in our own 
instructional and consultation efforts, we can achieve a higher level of 
measurement literacy by reintegrating measurement faculty and 
measurement knowledge into t , several reform movements in psychology 
and education. 
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Cla rify method? for asses sing prog ram comp liance mPa.;„rpmonf 
standards in ^cCTeditaHon mterla. The APA accreditation standards are 
currently under revision, both to widen the scope of accreditation for clinical, 
counseling and school programs, and to provide for minimal as well as 
exemplary levels of program criteria. Regardless of the criteria for levels of 
accreditation tiiat might be developed, it is reasonable to assume that 
instruction in measurement will continue to be required in every accredited 
psychology program approved by tiie APA acaeditation process, including 
those that would be approved only at a minimal level. After all, testing and 
measurement has almost a 100 year history in APA and in psychology. If 
measurement psychology colleagues partiduate actively in developing 
standards for professional psychology programs, tiiat include adequate 
measurement insti-uction, they will be furthering this longstanding history of 
measurement in psychology. 

Division 5, 15, and other interested groups of psychologists can 
participate directly in tiiis process by offering tiie technical assistance of 
measurement specialists who can assist in definitions and criteria for what 
would be considered to be a minimal level of insti-uction in measurement, 
and provide examples of ways tiiat an accreditation review panel can evaluate 
a program's measurement offerings. 

Participation in national confprpnrPQ on |>ro^^i ate education anr> p ncf- 
doctoral professional p reparaHnn fnr pr.^H-... January, 1990, tiie Assembly 
of Scientist-Practicioner Psychologists will sponsor a national conference on 
tiie training of sdentist-practitioner psychologists at tiie University of Florida 
in Gainesville. Division 5 and 15 can be sure that tiiese concerns are 
represented in this conference. At tiie very least the conference should 

'■ ''"©""ai-i^iattiuoner psycnology programs can 

capitalize on new developments in measurement theory and incorporate 
tiiem into their programs both in terms of faculty appointinents, as well as in 
course instruction. 

Reintegrate programs of me a surement instruction into p ropos ils fnr 
r eform in (he preparation of education p rofessionals. In the proposals for 
reform of tiie ways that teachers and other education professionals are 
prepared for their practice roles, it is essential tiiat measurement specialists, as 
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weu as educational psychologists, contribute a perspective to these discussions 
that currently seems not to be present. The newsletters of the Holmes Group, 
one of the leading national organizations organized to consider refon.>v in 
teacher education, composed of dean^ of schools of education, seem to reflect 
a perspective that centers attention on the conditions under which teachers 
teach, and ways lo promote interest in teaching, and less on the knowledge 
base necessary for instruction. Some members of the Holmes Group are 
leaders in Division 5, and in educational and psychological measurement 
generally. Their attention needs to focus on the knowledge necessary for 
educational practice, and in tiie case of measurement literacy, ths knowledge 
necessary to meet minimal standards for the appropriate selection, use, and 
interpretation of tests. We have a role to play here as well. 

Review the rnnfpnf nf n.H•.^no^ ^^n minations for psvcholn^ VaT 
educational prartitionprs The national examination used by aU state 
psychology licensure boards as wall as the national examinations for 
educational professionals can be reviewed to ascertain the extent to which 
measurement principles necessary for competent test use are integrated into 
the examinations. Taking into account the item writing competencies of 
many of our colleagues, we can expect tiiat they can provide examples of 
measurement content can be proposed for inclusion in revisions of these 
examinations. 

As new proposals for evaluating teacher compe.ance by performance 
on teacher examinations are developed, raising the level of measurement 
Kteracy can be approached by incorporating items reflecting acceptable levels 
of test user competence. 



Supply s ide pOSsibilitips 



.Create pio an4 pnsfdnctoral fellowh ips for p romki ng measuremenf 
students. Having available pre and postdoctoral fellowships to be awarded to 
candidates in those institutions offering a graduate program in psychomeh-ics 
might be a way of encouraging tiie interested students to select measurement 
as their chosen area of doctoral studies. Although some research and testing 
companies and State Boards of Education already have attempted this 
approach, combining the efforts of possible funding sources with 
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representative faculty at those instutitions with measurement programs, 
might create possibilities for improving the utilization of fellowships as a 
recuitment device, or to identify ways and places where the funds for student 
support might be best allocated. And in the event that those considering 
support of doctoral fellowships are concerned that the recipients might go on 
to careers in business raUier than back to institutions or the measurement 
community, a payback provision could be introduced that would require the 
recipient to commit a term of service after graduation as a condition for 
receiving the fellowship award. 

Endow chairs and increas e the measurement supp l y One strategy 
might be to negotiate with test publishers to set aside voluntarUy a portion of 
their profits to endow chairs in psychometrics on univereity campuses. Since 
the qualifications of the person holding the chair can be determined in part by 
the endowment, test publishers and psychometiic research institutes can 
have a direct impact on the future direction of measurement instruction and 
research. 

Give prizes for otif standing examp les of insh-nrfin nal modules in 
promote measurement literary amon^ tParTiPrg or prarHrinnprc inmy 
attempts to identify some of the problems associated with the demand and 
supply side of the measurement literacy matter, I do not intend to minimize 
the excellent work of professors and authors who have furthered this cause by 
their creation of excellent instructional materials. But formal recognition of 
these important applied measurement efforts might elevate their work to a 
platform where tangible and public recognition are available. 

Encourage state boards of ^dura tion to establish guidelines for tesHng 
specialists in srhon ls. All efforts to improve the measurement literacy of 
teachers and educational practitioners should be supported by recognition of 
the need for school district employees and others who are responsible for the 
use and interpretation of tests in schools to have demonsti-able competence in 
measurement. Negotiations witii state school officers and school board 
presidents are the places to begin a direct attack of controUing the supply of 
literate measurement specialists in the public sector. 
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These proposals pomt out the many opportunities currently available 
JT having a positive influence on the measurement competence of 
psychologists and educators. We can't expect test publishers to have sole 
responsibiUty for regulating test use by refusing to sell tests to users whose 
credentiak do not reflect user competence. We can't expect employers to 
have sole responsibiUty for regulating test user competence by selecting only 
teachers, education professionals, and psychologists who reflect acceptable 
levels of user competence in measurement. And we can't expect that 
professionals will "make" thrmselves competent unless we in tiie 
measurement fieid assist tiiem by defining minimal levels of measurjiment 
Hteracy necessary for various practice ar i, by creating consultation resources 
that identify tiiose who are wHling to coUaborate in reforming measurement 
knowledge levels and practices, and by identifying ways tiiat measurement 
competence can be enhanced. 

If you agree witii me tiiat tiie level of measurement literacy among 
educational and psychological professionak needs our immediate attention, 
tiien I hope tiiat you will also agree tiiat impro^dng this state of affairs cannot 
be left to cihers. It is essentially our job since we are tiie ones witii tiie 
knowledge tiiat is not getting utilized. 
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Why Worry? 

I contend that the future of measurement in psychology in the decades 
to come will be dependent on the availability of measurement faculty in our 
universities, the range of measurement offerings on our campuses, the 
standards for measurement literacy reflected in the preparation of 
psychological and educational professionals, in criteria for professional 
program accreditation, and standards for licenses and credentials for 
psychological and educational practice. 

Our active participation in some of the activities that I have proposed 
above will affect directly the current level of supply and demand for 
psychometridans and psychometrics. We should acknowledge the excellent 
work of our colleagues who contributed to the revised Standards, a nd the 
several other testing documents fliat have been published subsequentiy as 
weU as the current efforts of the Joint Committee on Testing Practices, and its 
subcommittees. But while they can develop the documents tiiat support 
raising the level of measurement literacy among education and psychology 
professionals, the documents will not lead us far toward the goal of 
measurement Uteracy, unless we, as measurement folks, carry the movement 
back to our places of work, as well as outward to the various committees 
involved in proposals for revising national standards in these fields 
dependent on measurement literacy for competent practices. 
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